The meaning of the interface : A functional and holistic evaluation of a meeting software system

The likelihood of a newly completed software system bringing desired advantages to organizational work depends, in part, upon the ability of users to recognize the functions and capabilities contained in the software. A preliminary evaluation attempts to assess the degree of match between designer and user perspectives of the software. Such an evaluation takes place after standard debugging and other error-correction procedures arc complete, but before the product is distributed for widespread use. In this paper we expound a preliminary evaluation approach for a group decision support system (GDSS) with the objective of contributing to the development of a standard evaluative procedure. The proposed approach consists of identifying and evaluating the GDSS in terms of its (1) functional, (2) holistic, and (3) interface characteristics. Applying this approach to a meeting software system, we were able to identify the strengths and weaknesses of the system independent of the task for which it was used. We believe the approach presented here provides a useful addition to the GDSS evaluation process and allows for a better understanding of why different software designs have certain effects on the decision makers who use them. Subject Areas: Decision Support Systems, Group Decision Support Systems, Software Design and Evaluation INTRODUCTION Evaluation of a newly completed software product is one of the more challenging activities in the systems development process. To a large extent, the issues involved in software evaluation are the same as those encountered with any technological innovation: Does it bring the benefits we, as designers, hoped for? Do users have sufficient understanding of the technology’s capabilities? What are users’ interpretations of the technology? Is less than optimal use of the system due to inadequacies in the product’s design? Will changes in the interface of the system encourage more effective use? For researchers interested in explaining how and why systems have certain effects on people, these questions take on enormous importance. Indeed, much of the mainstream of information systems research is concerned, to some dcgree, with the evaluation of software (Dickson & DeSanctis, 1990). There is particular interest in assessing the impacts of novel, leading-edge products and in whether these technologies are able to bring their promised “added value” to organizational work (Baily & Gordon, l988; Bowen, 1986; Edelman, 1981; Sharda et al., 1988). The evaluation process typically begins outside of the actual context in which the software is intended to be used (Card et al., 1983). Data from a preliminary evaluation can provide performance benchmarks on the quality of the interface, efficiency of processing, and overall user impressions of the system, which can be useful when modifying the product prior to its installation or anticipating the eventual success of the product once it is used in its intended context (Rubinstein & Hersh, 1984). Moreover, users’ early reactions to a new technology, particularity their interpretations of its role or meaning in a work context, can aid in explaining the eventual effects of the technology on organizational life. Alhough few would disagree with the importance of evacuating new software prior to its use for “live” purposes, it is less clear exactly how this preliminary evaluation process should be conducted. Although some basic principles and approaches may apply across systems, the evaluation must also cater to the particular software at hand. The criteria for assessing the effectiveness of a graphics package, for example, are somewhat different from those we might apply to an electronic mail system (see Baecker & Buxton, 1987). Consequently, as new forms systems emerge, unique benchmarks and procedures for conducting evaluations of those systems must be developed. In this paper we expound a preliminary evaluation approach for a group decision support system (GDSS). In recent years, there has been substantial interest in measuring the impacts of these systems when they are used in actual group meetings (see Erickson, 1989; Pinsonneault& Kraemer, 1989), but there have been few documented cases of interface assessments or preliminary evaluations. Several authors have described differences in interface designs across so-called ‘groupware’ products (Foster, 1986; Gray & Olfman, 1989); and difficulties in interface design have been blamed for early failures of groupware (Bjerknes & Bratteteig, 1988; Grudin, 1988). But, as Erikson (1989) points out, there arc many unresolved problems rcgarding wbat groupware products represent to those who use them -independent of a particular context-and until these are resolved, successful use of groupware may remain less than adequate. With this goal in mind, this paper outlines a rationale and approach for conducting a preliminary evaluation of a GDSS. Using principles developed by Card et al. (1983), Silver (1988), and DeSanctis and Gallupe (1987), and techniques prescribed by Roberts (1979), Roberts & Moran (1983), Card et al. (1983), and Rubinstein & Hersh (1984), wc compare two different versions, or models, of a GDSS in terms of their functionality, holistic attributes, and interface characteristics. Our analysis attempts to determine whether the two GDSS versions: (a) are recognized by users as containing appropriate groupware components (core tasks); (b) meet minimal standards for ease of use and user satisfaction, and (c) have inherent propensities to positively change the way in which face-to-be decision-rnaking meetings are conducted. We explore whether the two versions of the software differ with respect to these three dimensions. This paper does not prescribe a standard approach to preliminary evaluation of GDSSs; nor does